Chapter 2: Image Transformations

Introduction

In the previous section we looked at manipulating the pixels of an image by operations like blurring or edge detection. Now let's take a look at warping images using transformations. These processes don't modify the content of images, but instead deform them geometrically. OpenCV has a number of functions which do this. First we'll examine the sample which demonstrates the cv::warpAffine() function. Open up the CinderBlock sample located at blocks/openCV/samples/ocvWarp and run it. You will see a rotated and scaled version of the input image, which is a photograph by Trey Ratcliff.

warp_warp.jpg



Affine Warping

Cool - let's explore how this thing works. We'll start with setup:

void ocvWarpApp::setup()
{
mInputImage = ci::Surface8u( loadImage( loadResource( RES_IMAGE ) ) );
mRotationCenter = mInputImage.getSize() * 0.5f;
mRotationAngle = 31.2f;
mScale = 0.77f;
mParams = params::InterfaceGl( "Parameters", Vec2i( 200, 400 ) );
mParams.addParam( "Rotation Center X", &mRotationCenter.x );
mParams.addParam( "Rotation Center Y", &mRotationCenter.y );
mParams.addParam( "Rotation Angle", &mRotationAngle );
mParams.addParam( "Scale", &mScale, "step=0.1" );
updateImage();
}


This should all be familiar. We load our image from a resource and put it in mInputImage. Then we initialize some member variables which are the parameters of our warp: a mRotationCenter that is the center of the image, mRotationAngle of 31.2 degrees, and a mScale of 0.77. Then we build a params::InterfaceGl to create a GUI for these parameters. Last, we call updateImage(), which is where the interesting OpenCV work happens:

void ocvWarpApp::updateImage()
{
cv::Mat input( toOcv( mInputImage ) );
cv::Mat warpMatrix = cv::getRotationMatrix2D( toOcv( mRotationCenter ), mRotationAngle, mScale );
cv::warpAffine( input, output, warpMatrix, toOcv( getWindowSize() ), cv::INTER_CUBIC );
mTexture = gl::Texture( fromOcv( output ) );
}


The first two lines here also are familiar - we're just creating a cv::Mat called input which contains our mInputImage and then an empty cv::Mat to hold our output. The next line is new though. It creates a cv::Mat as well, not for holding an image, but the mathematical transform we want to apply to each pixel's position. As you may know, matrices are often used to express a series of geometric transformations. cv::getRotationMatrix2D() is a convenience method which creates the correct transformation matrix to achieve a rotation of degrees mRotationAngle around the point mRotationCenter, all preceeded by a scale of magnitude mScale.

In the next line, we make use of this matrix in our call to cv::warpAffine(). You can think of this routine as applying the matrix warpMatrix to each pixel in the input image, assigning it a new position in the output image. In reality, the exact inverse is what happens in order to prevent holes in the output image. OpenCV looks at each pixel in the output image and applies the inverse transformation to identify the source pixel in the input image. If this doesn't quite make sense yet, don't get too caught up in the details - just trust that taking the result of cv::getRotationMatrix2D() lets us build a cv::Mat we can use to warp the image using cv::warpAffine(). Note the final parameter for cv::warpAffine(), which is set to cv::INTER_CUBIC in the example above. This is the interpolation parameter, which (simplifying a bit) tells OpenCV how many surrounding pixels to consider when it calculates each output pixel. Other common values include cv::INTER_NEAREST, cv::INTER_LINEAR and cv::INTER_LANCZOS4. In general using more samples looks nicer (particularly when increasing the size of the image), but is slower. The zoomed screenshots below depict some of the interpolation modes:

warp_interp.png


Perspective Warping


Let's pause and consider the affine part of cv::warpAffine()'s name. An affine transformation is one made up of rotation, translation and scale (and technically shear, though that is less commonly used). Another way to think of this is that an affine transformation can transform a rectangle into any parallelogram. But what about less rigid transformations? For example, what if I wanted to "pull" the corner of a rectangle somewhere, but leave the other three corners in place? An affine transformation can't create this warp - we need a perspective transformation. Let's take a look at how to achieve this using OpenCV. Open up the CinderBlock sample located at blocks/openCV/samples/ocvPerspective. Go ahead and run the sample - you should see an image about like this one:

warp_persp.jpg


In the screenshot above we can see a perspective transformation in action, applied to a photograph by Pedro Szekely. Let's take a look at the source. Much of it is devoted to interacting with the mouse, allowing you to drag the corners around the window. It maintains an internal array of the four corners, called mPoints. The interesting bit is in ocvPerspectiveApp::updateImage():

void ocvPerspectiveApp::updateImage()
{
cv::Mat input( toOcv( mInputImage ) ), output;
src[0] = cv::Point2f( 0, 0 );
src[1] = cv::Point2f( mInputImage.getWidth(), 0 );
src[2] = cv::Point2f( mInputImage.getWidth(), mInputImage.getHeight() );
src[3] = cv::Point2f( 0, mInputImage.getHeight() );
for( int i = 0; i < 4; ++i )
dst[i] = toOcv( mPoints[i] );
cv::Mat warpMatrix = cv::getPerspectiveTransform( src, dst );
cv::warpPerspective( input, output, warpMatrix, toOcv( getWindowSize() ), cv::INTER_CUBIC );
mTexture = gl::Texture( fromOcv( output ) );
}


Just as cv::getRotationMatrix2D() generates the right matrix for cv::warpAffine(), for perspective transforms we use cv::getPerspectiveTransform() to generate a cv::Mat for cv::warpPerspective(). This function takes a two arrays of 4 cv::Point2f's as input. The first represents the original positions of four corners. In our case, these original points are the corners of our image, which we prepare in the src array, ordered in clockwise order starting with the upper-left corner (0,0). Next, we prepare the dst array, which stores the warped positions of the corresponding four corners. The sample allows users to drag these four corners around with the mouse (represented with the orange dots), so we simply convert these ci::Vec2f's into cv::Point2f's using ci::toOcv(). The call to cv::getPerspectiveTransform() returns the 3x3 matrix which warps src into dst. We then pass this matrix as input to cv::warpPerspective(), whose parameters are identical to those of cv::warpAffine(). Keep in mind that the resulting cv::Mat is still just a rectangular image, but filled with black everywhere that is outside of the warped input image.

Exercises

  1. Checkout the cv::resize() routine. How does its performance compare to cv::warpAffine()?
  2. How does cv::warpPerspective() compare to the texture mapping in OpenGL? Build an app that matches as closely as possible - you can use CameraPersp::worldToScreen() to determine the coordinates for cv::getPerspectiveTransform().