Hardware

The hardware portion of the vision system consists of a Logitech QuickCam Pro 5000 mounted above the robotic arm. The mount is a simple three legged structure made of irrigation tubing that is attached to the wooden base board via plugs for the irrigation tubing that are screwed into the board. The photo shows the relevant part of the structure:

Software

The code that analyzes the software leverages the OpenCV computer vision library through the Emgu CV cross platform .Net wrapper. In order to be useful the vision system first needs to be calibrated so that a pixel in the image can be associated with a physical location on the base board. This is accomplished by lining up a simple grid (CameraCalibrationGrid.pdf) on the base board and then detecting the squares in the image using OpenCV's FindContours function. Below is a screenshot of the Camera Calibration application that performs the calibration. The CameraCalibration project is part of the solution Vision.sln in the Vision folder (http://code.google.com/p/robotic-tic-tac-toe-lynxmotion/source/browse/trunk/#trunk/Vision).

The app shows three versions of the live camera image (cropped). On the left is the original image, the middle shows an inverted black and white version of the original image with thickened lines, and the right image shows the centers of the detected rectangles. In order to verify the order of the detected rectangles the centers are drawn with increasing pen widths. The app expects to find 22 rectangles (one surrounding rectangle and 3 x 7 small rectangles). The Capture Calibration button is only enabled when the app has detected 22 rectangles. Once the calibration is captured the calibration data needs to be saved (menu File / Save).

The detection of rectangles is performed in the class MainFormModel.cs. The relevant code blocks are:

Image Manipulation

public void ProcessFrame(int threshold)
{
    m_OriginalImage = m_Capture.QueryFrame();

    m_ClippedImage = m_OriginalImage.Copy(this.RegionOfInterest);

    // Make the dark portions bigger
    m_ErodedImage = m_ClippedImage.Erode(1);

    //Convert the image to grayscale
    m_GrayImage = m_ErodedImage.Convert<Gray, Byte>();

    m_BlackAndWhiteImage = m_GrayImage.ThresholdBinaryInv(new Gray(threshold), new Gray(255));

    FindRectangles(m_BlackAndWhiteImage);

    this.FoundRectangleCount = m_FoundRectangles.Count;
    if (this.FoundRectangleCount == m_ImageModel.ExpectedRectangleCount)
    {
        m_ImageModel.AssignFoundRectangles(m_FoundRectangles);
        m_FoundRectanglesImage = CreateRectanglesImage(m_ImageModel.GetInsideRectangles());
    }
    else
    {
        m_FoundRectanglesImage = CreateRectanglesImage(m_FoundRectangles);
    }
}

Rectangle Detection

private void FindRectangles(Image<Gray, Byte> blackAndWhiteImage)
{
    m_FoundRectangles.Clear();

    using (MemStorage storage = new MemStorage()) //allocate storage for contour approximation
    {
        for (Contour<Point> contours = blackAndWhiteImage.FindContours(
            Emgu.CV.CvEnum.CHAIN_APPROX_METHOD.CV_CHAIN_APPROX_SIMPLE,
            Emgu.CV.CvEnum.RETR_TYPE.CV_RETR_LIST,
            storage);
            contours != null;
            contours = contours.HNext)
        {
            Contour<Point> currentContour = contours.ApproxPoly(contours.Perimeter * 0.05, storage);
            //Debug.WriteLine(currentContour.Area);

            if (currentContour.Area > 250) //only consider contours with area greater than 250
            {
                if (currentContour.Total == 4) //The contour has 4 vertices.
                {
                    if (IsRectangle(currentContour))
                    {
                        m_FoundRectangles.Add(currentContour.GetMinAreaRect());
                    }
                }
            }
        }
    }
}

/// <summary>
/// Determines whether the angles are close enough to 90 degrees
/// </summary>
/// <param name="contour"></param>
/// <returns></returns>
private bool IsRectangle(Contour<Point> contour)
{
    Point[] pts = contour.ToArray();
    LineSegment2D[] edges = PointCollection.PolyLine(pts, true);

    for (int i = 0; i < edges.Length; i++)
    {
        LineSegment2D currentEdge = edges[i];
        LineSegment2D nextEdge = edges[(i + 1) % edges.Length];

        double angle = Math.Abs(nextEdge.GetExteriorAngleDegree(currentEdge));
        if (angle < 80 || angle > 100)
        {
            return false;
        }
    }

    return true;
}

Capturing Calibration Information

When the Capture Calibration button is pressed the association between the rectangle centers in the camera image and the physical coordinates (CameraVsPhysicalPoint array) is retrieved from the CalibrationImage class (CalibrationImage.cs) which is part of the Vision project.

public void CaptureCalibration()
{
    Debug.Assert(CaptureIsPossible);
    CameraVsPhysicalPoint[,] cameraVsPhysicalPoints = m_ImageModel.GetCameraVsPhysicalPoints();
    //Print(cameraVsPhysicalPoints);

    Vision.CameraCalibration.Instance.Initialize(cameraVsPhysicalPoints);
}

This calibration information is used to initialize the CameraCalibration singleton (second to last line in the code block above). The CameraCalibration class (CameraCalibration.cs) is also used to store the calibration data in a text file or read it from a text file. The main purpose of the class is to calculate the physical coordinates for any camera pixel by interpolating between the measured calibration points.