Italiano - English

How to display an OpenCV image or video in your own MFC interface

To display an OpenCV image or video into your own MFC interface in Visual C++ you can use GDI's function but keep care of details.

Basics

OpenCV Mat is a complex class but it provides the pointer to memory block where the image data is stored (see How the image matrix is stored in the memory)

GDI defined in MFC framework, provides such functions like StretchDIBits() and  SetDIBitsToDevice() to draw an image stored in memory directly into a Device Context (DC). See How DIB bitmap is stored in the memory:

In Windows, each GUI control has its own DC available via GetDC() methods.

The idea it's easy:

  • Drop down a control into your GUI (Static Text or a Picture) and map it into a member CStatic variable
  • Each time you want to display an OpenCV image, get the pointer to image data and draw the image into DC of your control.

Details

The idea is confirmed to be easy if we keep in mind that:

  • The GDI Bitmap uses stride(padding) on row bytes that must be DWORD (4 byte) aligned.
  • The GDI Bitmap manage only BGR(A) images. To show grey scale you have to provide a palette in the BITMAPHEADER.
  • The GDI Bitmap coordinates are bottom to top while OpenCV use Top to Bottom
  • In OpenCV the colour image matrix is stored always as BGR(A)
  • The OpenCV image might be not continuous (ROI or sub matrix)
  • The OpenCV image cols might be odd event anyway not DWORD aligned

Learn by example with the function: CvMatToWinControl

Ok we want a function that draws a cv::Mat into a CStatic MCF control so we start from here:

 
void CvMatToWinControl(const cv::Mat& img, CStatic* WinCtrl)
{
  if (WinCtrl == NULL || img.empty())
    return;
 
  int bpp = 8 * img.elemSize();
  assert((bpp == 8 || bpp == 24 || bpp == 32));
 
  //Get DC of your win control
  CClientDC hDC(WinCtrl);
 
  // This is the rectangle where the control is defined
  // and where the image will appear
  RECT rr;
  WinCtrl->GetClientRect(&rr);
  //rr.top AND rr.left are always 0
  int rectWidth = rr.right;
  int rectHeight = rr.bottom; 
...

The above code is simple, just remember that GetClientRect returns coordinates relative to the win control so left and top are always 0.

DWORD alignment and Continuous Matrix

We can fit both requirements using cv::copyMakeBorder function.

Low level GDI bitmap drawing functions require to know a memory block where the bitmap is stored. A cv::Mat provides the attribute data that is a pointer to memory block where the bitmap is stored, that is OK.

But cv::Mat can be a sparse matrix onr not continuous (sub matrix, ROI,...) in this case the bitmap isn't stored in unique memory block and we can't use data pointer as reference for GDI. If cv::Mat isn't continuous we need to copy source Mat into a new Mat so the new mat will be continuous.

DWORD alignment is related to number of bytes used by each rows. A RGBA (32 bit) image is always DWORD aligned because each pixel requires 4 byte.

Grey or RGB should be aligned. Because we don't want to break pixel memory to gain DWORD alignment at byte level, we will add needed columns to gain DWORD at pixel level.

Suppose you have a RGB image 1013cols. In OpenCV this requires 1013*3=3039 bytes each row. Adding 1 byte to each row we'll gain DWORD align. But, how we can add 1 byte to each row in OpenCV if the image is RGB? I prefer using cv::copyMakeBorder to add 3 pixel each row and gain DWORD align at pixel level. This solves Continuous image issues too;
 
...
  ///------------------------------------
  /// DWORD ALIGNMENT AND CONTINOUS MEMORY
  /// The image must be padded 4bytes and must be continuous
 
  int padding = 0;
  //32 bit image is always DWORD aligned because each pixel requires 4 bytes
  if (bpp < 32)
    padding = 4 - (img.cols % 4);
 
  if(padding==4)
    padding = 0;
 
  cv::Mat tmpImg;
  if (padding > 0 || img.isContinuous() == false)
  {
    // Adding needed columns on the right (max 3 px)
    cv::copyMakeBorder(img, tmpImg, 0, 0, 0, padding, cv::BORDER_CONSTANT, 0);
  }
  else
  {
    tmpImg = img;
  }
...
 

We will draw tmpImg that is Continuous and DWORD aligned. Additional care will be used to avoid to draw the border.

Prepare the BITMAPINFO

GDI functions require a BITMAPINFO to know info about the source image we want to draw. BITMAPINFO structure has an header and a pointer to a colour palette. In case your image is RGB(A) you have to prepare just the header.

To display a grey scale image using GDI we have to prepare a BITMAPHEADER that contains an uniform palette for RGB. This means that GDI still consider the image as RGB but it shows using a grey palette.

Because BITMAPINFO contains just a pointer to the palette we prefer to create an unique memory block for BITMAPINFO that contains both header and palette. So we are using a buffer. Let me show here:

 
...
  ///----------------------
  /// PREPARE BITMAP HEADER
  /// The header defines format and shape of the source bitmap in memory
 
  // extra memory space for palette -> 256*4
  uchar buffer[sizeof(BITMAPINFO) + 256*4];
  BITMAPINFO* bmi = (BITMAPINFO*)buffer;
  BITMAPINFOHEADER* bmih = &(bmi->bmiHeader);
  memset(bmih, 0, sizeof(*bmih));
  bmih->biSize = sizeof(BITMAPINFOHEADER);
  bmih->biWidth = tmpImg.cols;
  bmih->biHeight = -tmpImg.rows;// DIB are bottom ->top
  bmih->biPlanes = 1;
  bmih->biBitCount = bpp;
  bmih->biCompression = BI_RGB;
 
  //Sets the palette only if image is grey scale
  if (bpp == 8)
  {
    RGBQUAD* palette = bmi->bmiColors;
    for (int i = 0; i < 256; i++)
    {
      palette[i].rgbBlue = palette[i].rgbGreen = palette[i].rgbRed = (BYTE)i;
      palette[i].rgbReserved = 0;
    }
  }
...
 

Image flipped because of reverse coordinate

This issue has very easy solution given in the definition of BITMAPINFOHEADER. You need just to use negative number of rows:

 
bmih->biHeight = - tmpImg.rows; /* look at negative sign to solve flip issue */
 

Drawing to DC

Ok there are different way to do this. We prefer to use SetDIBitsToDevice() or StretchDIBits(). These functions copy a bitmap from a memory block into a DC rectangle. Name of the functions explains that you can just copy or copy and adapt.

You can proceed as below:

 
...
  /// -----------
  /// Draw to DC
 
  if (tmpImg.cols == rectWidth  && tmpImg.rows == rectHeight)
  {
    // source and destination have same size
    // tranfer memory block
    // NOTE: the padding border will be shown here. Anyway it will be max 3px width
 
    SetDIBitsToDevice(hDC,
      //destination rectangle
      0, 0, rectWidth, rectHeight,
      0, 0, 0, tmpImg.rows,
      tmpImg.data, bmi, DIB_RGB_COLORS);
  }
  else
  {
    // Image is bigger or smaller than into destination rectangle
    // we use stretch in full rect
 
    // destination rectangle 
    int destx    = 0, desty = 0;
    int destw    = rectWidth;
    int desth    = rectHeight;
 
    // rectangle defined on source bitmap
    // using imgWidth instead of tmpImg.cols will ignore the padding border
    int imgx    = 0, imgy = 0;
    int imgWidth  = tmpImg.cols - padding;
    int imgHeight  = tmpImg.rows;
 
    StretchDIBits(hDC,
      destx, desty, destw, desth,
      imgx, imgy, imgWidth, imgHeight,
      tmpImg.data, bmi, DIB_RGB_COLORS, SRCCOPY);
  }
} // end function

Set... function is faster than Stretch... while with Stretch... you can scale your image over destination rectangle.

All in one with the PkMatToGDI Class

This class provides a complete method to draw an OpenCV Mat image directly into MFC Gui. It's optimised to display a feed from video (cam or file) and it can be used to display a single cv::Mat.

It uses GDI to transfer the image from OpenCV memory to DC of a CStatic Control like a Picture or Static Text with auto fitting and stretch.

Only Gray,RGB, and RGBA images are supported.

ThreadSafe

You can use this class also in a worker threads only if:

  • two threads can't have DCs to the same window at same time
  • two threads can't try to manipulate the same DC at same time

Hints

Windows GDI requires DWORD alignment for rows and continuous memory block for the source bitmap.

You can forget this requirements because the class checks the requirements and creates a right temporary image in case is needed (loosing a bit of time).

If you provide continuous image with modulo 4 columns you will reduce time and memory for drawing.

To improve general memory management is strongly suggested to use always images where cols %4 but, if given image is not continuous the class will create the temp image again.

Finally the class isn't memory consuming because it uses an internal cv::Mat to recycle always it's possible. This is very useful when you are rendering a video where all frames have almost same size.

See also

There is an interesting implementation of drawing into GDI DC in WebRTC (Web-based real-time communication) by Google. See gdivideorenderer for more details https://code.google.com/p/webrtc/source/browse/trunk/talk/media/devices/gdivideorenderer.cc?r=5038

Another nice solution is the MFC GUI of the Background Subtraction Library. The BGSLibrary (https://github.com/andrewssobral/bgslibrary) was developed by Andrews Sobral and provides an easy-to-use C++ framework based on OpenCV to perform background subtraction (BGS) in videos. The MFC version of their test application implements nice worker thread and OpenCV video drawing into a Windows GUI.

Vote this page:

6 Comments:

#1 Sent by bob 03-06-2015

Nice

#2 Sent by Eddy 04-01-2016

Memory leak when using this class.

#3 Sent by PkLab 07-01-2016

@Eddy: I'll investigate

#4 Sent by tuvi 05-02-2016

Thank you so much. I am new to Opencv and MFC. It helped me a lot.

#5 Sent by eddievid1990 11-01-2017

Very interesting site i have bookmarked pklab.net for future reference.

#6 Sent by Dan Bloomquist 22-01-2017

PkMatToGDI just works! No Pain at all. Thanks much, so better than reinventing the wheel.
The coding examples presented here are for illustration purposes only. The author takes no responsibility for end-user use
This work is property of Pk Lab. You can use it for free but you must retain author's copyright.