If you're like me, you prefer to learn by example, so let's jump right into looking at some code. We'll begin by walking through a sample included with the OpenCV CinderBlock, located at blocks/opencv/samples/ocvBasic. This sample loads an image as a resource, and then performs some basic image processing and displays the results. Let's examine the code in detail:
#include "CinderOpenCV.h"
Here's the first interesting line. We're just including the OpenCV CinderBlock's header file. This file contains the glue between Cinder and OpenCV, and it in turn includes the relevant headers from OpenCV itself. Next let's look at the setup() method:
void ocvBasicApp::setup() { ci::Surface8u surface( loadImage( loadResource( RES_IMAGE ) ) ); cv::Mat input( toOcv( surface ) ); cv::Mat output; cv::medianBlur( input, output, 11 ); ... mTexture = gl::Texture( fromOcv( output ) ); }
The first line should be familiar. We're just creating a ci::Surface by loading an image from a resource. Next, we dive right into some proper OpenCV code. We create two instances OpenCV's cv::Mat class. This class is very similar to Cinder's Surface, and can be used to represent an image. More on this important class later. Notice the constructor parameter for input - the result of a call to toOcv() with the surface we loaded earlier. This is our first exposure to the CinderBlock layer. This function simply translates a Cinder class into a form that is usable directly with OpenCV - in this case a ci::Surface into something suitable to construct a cv::Mat with. We'll get deeper into how the OpenCV CinderBlock works in a bit. The second portion of the line constructs an empy cv::Mat called output.
Next, we get into some actual image processing. In this case, we are using OpenCV's medianBlur() function, which smooths an image using a median filter. The first parameter is the source cv::Mat, input in our case. The second parameter is the cv::Mat which will hold the result. OpenCV automatically creates a cv::Mat which is of the appropriate size and depth, and in our case will put this result into output. The final parameter is how large a window is examined for computing the median of each pixel - we've selected a kernel size of 11
.
The last line of setup() creates an OpenGL texture used later in draw(). It makes use of the CinderBlock's fromOcv() function, which behaves just like toOcv() does but in the other direction, converting OpenCV's data structures into Cinder's equivalents. Running the sample gives us a smoothed version of our input image, which is a photograph by Trey Ratcliff.
Nice. So let's try tweaking this code a bit. Try changing the kernel size from 11
to something like say, 21
. Just make sure it's odd and bigger than 1
. How do I know that? Well, the OpenCV docs told me. Now's a good time to get familiar with what they look like - checkout the description of medianBlur(). As you learn OpenCV, you'll want to familiarize yourself with these docs, but for now let's keep moving.
Let's experiment with a different image processing operator. First, comment out the line in setup() which calls cv::medianBlur(), and replace it with a call to the Sobel edge detection operator:
The first two parameters to cv::Sobel should be familiar - they're the same as cv::medianBlur(). The third parameter is a little different though. Instead of automatically deriving what kind of cv::Mat the output should be from the input, cv::Sobel() wants us to tell it what kind of image we want. By passing CV_8U
we have asked for an 8-bit unsigned image, which is normal for most image processing. If we wanted say, a floating point image, we could pass CV_32F
instead. The fourth and fifth parameters, which are xOrder and yOrder respectively, allow us to select between horizontal and vertical edges. By passing 0
and 1
, we have selected the vertical edges. Run this, and you should see a result about like this:
There are additional optional parameters to cv::Sobel(). Checkout its documentation here. How do the horizontal edges compare? Try swapping the 0
and 1
, or experiment with the kernel size parameter the docs mention.
And now we'll experiment with a final image processing operator, cv::threshold(). The first and second parameters are consistent with cv::Sobel() and cv::medianBlur() - just the input and output image represented as cv::Mat's. The third parameter specifies what value defines above and below, and the fourth is the maximum value. Since we're working on 8-bit unsigned images, we'll pass 128
and 255
for these two values.
cv::threshold( input, output, 128, 255, CV_8U );
The cv::Mat class provides a representation of an image, much like Cinder's Surface or Channel classes. To determine the size of a cv::Mat, use code like this:
cv::Mat myMat( ... ); console() << "myMat is " << myMat.size().width << " x " << myMat.size().height << " pixels." << std::endl;
The design of cv::Mat is also quite similar to ci::Surface or ci::Channel with respect to memory management. Like these Cinder classes, cv::Mat maintains a reference count, automatically freeing the associated memory when the refence count drops to zero. Furthermore, assigning one cv::Mat to another is fast, since no image data is copied - just pointers to image data. This means you're safe passing cv::Mat's by value (again, just like ci::Surface or ci::Channel).
cv::Mat matA( toOcv( "C:\\imageA.png" ) ); cv::Mat matB = matA; // matB and matA both reference the same image cv::Mat matC( toOcv( "C:\\imageC.png" ) ); matC = matA; // the image matC was constructed with gets freed automatically here
Also just like Cinder's equivalent classes, you can allocate a cv::Mat which is empy - the equivalent of a null pointer. To test for null with ci::Surface, we write something about like
Surface mySurface( ... ); if( mySurface ) ... do something with non-null mySurface ...
To perform the equivalent null-test with cv::Mat, use code like this:
cv::Mat myMat( ... ); if( myMat.data() ) ... do something with non-null myMat ...
A key difference between cv::Mat and Surface or Channel is that a cv::Mat can contain an image of many different varieties. While say, Cinder's Surface8u class always represents an RGB(A) 8-bit image, a cv::Mat might represent a 32-bit floating point RGB image, or an 8-bit grayscale image, or perhaps a 16-bit YUV image. To determine what sort of data the cv::Mat contains, use cv::Mat::type(). This will return a value we can compare against constants like CV_8U3
, which implies an 8-bit unsigned, 3-color image (corresponding to a ci::Surface8u with no alpha channel), or say, CV_32F1
, which corresponds to a ci::Channel32f. These constants take the form CV_
<bit-depth>{U
| S
| F
}<number-of-channels>, and are discussed in detail here.
if( input.type() == CV_8U3 ) ... do something with a 3 channel, 8-bit image ... else if( input.type() == CV_32F3 ) ... do something with a 3 channel, 32-bit float image ... else ... fail gracefully ...
The OpenCV CinderBlock has been designed to offer a combination of performance, flexibility and convenience. Rather than imposing an additional API users must learn and which must be maintained as OpenCV itself evolves, the CinderBlock consists of essentially just two overloaded functions, ci::toOcv(), and ci::fromOcv(). Armed with these, you can easily exchange data structures between OpenCV and Cinder. We have seen both used in the example code above:
ci::Surface8u surface( loadImage( loadResource( RES_IMAGE ) ) ); cv::Mat input( toOcv( surface ) ), output; ... mTexture = gl::Texture( fromOcv( output ) );
Note that the first two lines could be condensed into one:
cv::Mat input( toOcv( loadImage( loadResource( RES_IMAGE ) ) ) ), output;
Note also that we can pass the result of ci::fromOcv(cv::Mat) to functions like writeImage directly:
#include "cinder/ImageIo.h" writeImage( "openCV_output.png", fromOcv( output ) );
If you've worked with OpenCV in the past, particularly older versions, you may be familiar with its procedural API. In more recent releases, OpenCV has introduced an object-oriented C++ API which allows greater brevity and is type-safe. The OpenCV CinderBlock is based upon this more modern API. However if you are working with legacy code, or you simply prefer it, you can still call the procedural API, passing cv::Mat instances in place of IplImages
or Arr*
. For example, the two calls below are equivalent:
cv::Mat input( ... ), output; cvLaplace( input, output, CV_8U ); // call the C function cv::Laplacian( input, output, CV_8U ); // call the C++ function
A helpful introduction to the C++ API is available here.
1. Experiment with combining these basic OpenCV operators. How does the order of operations affect things? What happens if you run a threshold operator on top of a Sobel operator? What about a Sobel operator on top of a threshold operator?
2. Try making one of the image processing parameters (like the value parameter for cv::Threshold) user-controllable using params::InterfaceGl. If you've never made use of this feature of Cinder, start with the sample in cinder/samples/paramsBasic.
3. Adapt this sample to work based on the real-time input of a webcam using Capture, or the frames of a QuickTime movie using qtime::MovieSurface.