[FIXED] Trying to detect blue color from image using opencv, and getting unexpected result

Issue

I am new to OpenCV4Android. Here is some code I wrote to detect blue colored blob in an image. Among the following images, image 1 was in my laptop. I run the application and the frame captured by the OpenCV camera is image 2. You can look at the code to see what the rest of the images are. (As you can see in the code, all the images are saved in the SD card.)

I have the following questions:.

• Why hass the color of the light-blue blob turned out to be light-yellow in the rgba frame captured by the camera (shown in image 2).

• I created a boundingRect around the largest blue colored blob, but then ROI by doing rgbaFrame.submat(detectedBlobRoi). But you can see in the last image, it just looks like a couple of grey pixels. I was expecting the blue colored sphere separated from the rest of the image.

What am I missing or doing wrong?

CODE:

private void detectColoredBlob () { 
        Highgui.imwrite("/mnt/sdcard/DCIM/rgbaFrame.jpg", rgbaFrame);//check
        Mat hsvImage = new Mat(); 
        Imgproc.cvtColor(rgbaFrame, hsvImage, Imgproc.COLOR_RGB2HSV_FULL);
        Highgui.imwrite("/mnt/sdcard/DCIM/hsvImage.jpg", hsvImage);//check

        Mat maskedImage = new Mat(); 
        Scalar lowerThreshold = new Scalar(170, 0, 0); 
        Scalar upperThreshold = new Scalar(270, 255, 255); 
        Core.inRange(hsvImage, lowerThreshold, upperThreshold, maskedImage);
        Highgui.imwrite("/mnt/sdcard/DCIM/maskedImage.jpg", maskedImage);//check

        Mat dilatedMat= new Mat(); 
        Imgproc.dilate(maskedImage, dilatedMat, new Mat() ); 
        Highgui.imwrite("/mnt/sdcard/DCIM/dilatedMat.jpg", dilatedMat);//check
        List<MatOfPoint> contours = new ArrayList<MatOfPoint>();
        Imgproc.findContours(dilatedMat, contours, new Mat(), Imgproc.RETR_LIST, Imgproc.CHAIN_APPROX_SIMPLE);
        //Use only the largest contour. Other contours (any other possible blobs of this color range) will be ignored.
        MatOfPoint largestContour = contours.get(0);
        double largestContourArea = Imgproc.contourArea(largestContour);
        for ( int i=1; i<contours.size(); ++i) {//NB Notice the prefix increment.
            MatOfPoint currentContour = contours.get(0);
            double currentContourArea = Imgproc.contourArea(currentContour);
            if (currentContourArea > largestContourArea) {
                largestContourArea = currentContourArea;
                largestContour = currentContour;
            }
        }

        Rect detectedBlobRoi = Imgproc.boundingRect(largestContour);
        Mat detectedBlobRgba = rgbaFrame.submat(detectedBlobRoi);
        Highgui.imwrite("/mnt/sdcard/DCIM/detectedBlobRgba.jpg", detectedBlobRgba);//check
     }

1. Original Image in computer, This was captured by placing phone's camera infront of laptop screen.

2. rgbaFrame.jpg

3. hsvImage.jpg

4. dilatedImage.jpg

5. maskedMat.jpg

6. detectedBlobRgba.jpg

---

EDIT:

I just used Core.inRange(hsvImage, new Scalar(0,50,40), new Scalar(10,255,255), maskedImage);//3, 217, 225 --- 6, 85.09, 88.24 ...... 3 219 255, and I captured a screeshot of the website colorizer.org by giving it a custom HSV values for red color, i.e. for the OpenCV red Scalar(3, 217, 255) (which falls in the range set in the given inRange function, I scaled the channel values to the scale of colorizer.org, which is H=0-360, S=0-100, V=0-100, by multiplying H value by 2, and dividing both the S and V values by 255 and multiplying by 100. This gave me 6, 85.09, 88.24 which I set on the website, and took a screenshot (the first in the following images).

1. Original screenshot, I captured this frame.

2. rgbaFrame.jpg

3. hsvImage.jpg

4. maskedImage.jpg

5. dilatedMat.jpg

6. detectedBlobRgba.jpg

---

IMPORTANT:

The method given is actually invoked in my test application when I touch inside the rgbaFrame (i.e. it is invoked inside onTouch method). I am also using the following code to print to a TextView the Hue, Saturation, and Value values of the colored blob that I have touched. When I run this application, I touched the red colored blob, and got the following values: Hue:3, Saturation:219, Value:255.

public boolean onTouch(View v, MotionEvent motionEvent) { detectColoredBlob(); int cols = rgbaFrame.cols(); int rows = rgbaFrame.rows();

int xOffset = (openCvCameraBridge.getWidth() - cols) / 2;
int yOffset = (openCvCameraBridge.getHeight() - rows) / 2;

int x = (int) motionEvent.getX() - xOffset;
int y = (int) motionEvent.getY() - yOffset;

Log.i(TAG, "Touch image coordinates: (" + x + ", " + y + ")");//check

if ((x < 0) || (y < 0) || (x > cols) || (y > rows)) { return false; }

Rect touchedRect = new Rect();
touchedRect.x = (x > 4) ? x - 4 : 0;
touchedRect.y = (y > 4) ? y - 4 : 0;
touchedRect.width = (x + 4 < cols) ? x + 4 - touchedRect.x : cols - touchedRect.x;
touchedRect.height = (y + 4 < rows) ? y + 4 - touchedRect.y : rows - touchedRect.y;
Mat touchedRegionRgba = rgbaFrame.submat(touchedRect);

Mat touchedRegionHsv = new Mat();
Imgproc.cvtColor(touchedRegionRgba, touchedRegionHsv, Imgproc.COLOR_RGB2HSV_FULL);

double[] channelsDoubleArray = touchedRegionHsv.get(0, 0);//**********
float[] channelsFloatArrayScaled = new float[3];
for (int i = 0; i < channelsDoubleArray.length; i++) {
    if (i == 0) {
        channelsFloatArrayScaled[i] = ((float) channelsDoubleArray[i]) * 2;// TODO Wrap an ArrayIndexOutOfBoundsException wrapper
    } else if (i == 1 || i == 2) {
        channelsFloatArrayScaled[i] = ((float) channelsDoubleArray[i]) / 255;// TODO Wrap an ArrayIndexOutOfBoundsException wrapper
    }
}

int androidColor = Color.HSVToColor(channelsFloatArrayScaled);

view.setBackgroundColor(androidColor);
textView.setText("Hue : " + channelsDoubleArray[0] + "\nSaturation : " + channelsDoubleArray[1] + "\nValue : "
        + channelsDoubleArray[2]);

touchedRegionHsv.release();
return false; // don't need subsequent touch events 

}

Solution

There are multiple traps in converting an image to HSV color space and using HSV color space.

1. OpenCV uses a compressed hue range because original, hue ranges from 0 to 360 which means that the values can't fit in 1 byte (values 0 to 255) while saturation and value channels are exactly covered by 1 byte. Therefore, OpenCV uses hue values divided by 2. So the hue channel will be covered by matrix entries between 0 and 180. Regarding this, your hue range from 170 to 270 should be divided by 2 = range 65 to 135 in OpenCV.

2. hue tells you about the color tone, but saturation and value are still important to reduce noise, so set your threshold to some minimum saturation and value, too

3. very important: OpenCV uses BGR memory ordering for rendering and image saving. This means that if your image has RGB(a) ordering and you save it without color conversion, you swap R and B channels, so assumed red color will become blue etc. Unfortunately normally you can't read from the image data itself, whether it is RGB or BGR ordered, so you should try to find it out from the image source. OpenCV allows several flags to convert either from RGB(A) to HSV and/or from BGR(A) to HSV, and/or from RGB to BGR etc, so that is no problem, as long as you know which memory format your image uses. However, displaying and saving always assumes BGR ordering, so if you want to display or save the image, convert it to BGR! HSV values however will be the same, no matter whether you convert a BGR image with BGR2HSV or whether you convert a RGB image with RGB2HSV. But it will have wrong values if you convert a BGR image with RGB2HSV or a RGB image with BGR2HSV... I'm not 100% sure about Java/Python/Android APIs of openCV, but your image really looks like B and R channels are swapped or misinterpreted (but since you use RGBA2HSV conversion that's no problem for the hsv colors).

about your contour extraction, there is a tiny (copy paste?) bug in your code that everyone might observe once in a while:

MatOfPoint largestContour = contours.get(0);
    double largestContourArea = Imgproc.contourArea(largestContour);
    for ( int i=1; i<contours.size(); ++i) {//NB Notice the prefix increment.
        // HERE you had MatOfPoint currentContour = contours.get(0); so you tested the first contour in each iteration
        MatOfPoint currentContour = contours.get(i);
        double currentContourArea = Imgproc.contourArea(currentContour);
        if (currentContourArea > largestContourArea) {
            largestContourArea = currentContourArea;
            largestContour = currentContour;
        }
    }

so probably just this has to be changed to use i instead of 0 in the loop

MatOfPoint currentContour = contours.get(i);

Answered By - Micka
Answer Checked By - David Goodson (JavaFixing Volunteer)