|
||||||
|
WHY when writting image processing algorithms using OpenCV Mat.get in a image scanning loop is it so much slower than calling an equivalent OpenCV.processing_method?????
................ too many OpenCV calls is expensive (slows down system)
WHY? you are making a JNI call from your Android Java (similar issue in ios Swift) down to the native C++ library...and this is computationally expensive.
So this is why Imgproc.threshold() takes way less time than you cycling through the array with Mat.get(r,c).
-
Imgproc.threshold() is 1 JNI call
-
YOUR CODE = 1 Million JNI calls
-
for(r=all rows and c= all columns) { threshold on Mat.get(r,c) } --> is 1Million JNI calls for a 1,000x1,000 pixel image.
-
Solution to help (goes from 1,000,000 JNI calls to 2 JNI calls)
1)STEP1: convert Mat to a Java array
//step 1 perform laplacian on gray image using DOG operator 3x3 and also threshold at same time
// first convert to Java array to make processing faster
byte [] image = new byte[(int) (imageMat.total() * imageMat.channels())]; //size is num pixels * number fields (1=gray and 4=rgba]
imageMat.get(0,0,image);
2)STEP 2: do your processing on the Java array
EXAMPLE 1
// if wanted to visit each pixel in image as row, col would write
int width = gray.width();
int height = gray.height();
int num_pixels = (int) gray.total();
int channels = gray.channels();
for (int r =0; r<height; r++)
for (int c = 0; c<width; c++)
for(int i=0; i<channels; i++) //in case color channels =3 otherwise for gray =1
{
//to get pixel corresponding to row r and column c
if (image[r*(width*channels) + c*channels + i] < threshold)
image[r*(width*channels) + c*channels + i] = 0;
else
image[r*(width*channels) + c*channels + i] = (byte) 255;
}
EXAMPLE 2
// if simply wanted to visit each pixel in image and not index by row, column
int num_pixels = (int) gray.total();
int channels = gray.channels();
//if simply visiting each pixel and not visiting neighbors where need to know row, column
for(int index = 0; index<num_pixels; index++ )
for (int i=0; i<channels; i++) //in case color channels =3 otherwise for gray =1
{
//to get pixel corresponding to row r and column c
if (image[index*channels + i] < threshold)
image[index*channels + i] = 0;
else
image[index*channels + i] = (byte) 255;
}
3)STEP 3: create Mat from the processed Java array
EXAMPLE //place back into a Mat for OpenCV calls
imageMat.put(0, 0, image);
************WARNING**********
Be aware that Java does not have any unsigned byte data type, so be careful when working with it. The safe procedure is to cast it to an integer and use the And operator (&) with 0xff. A simple example of this would be
int unsignedValue = myUnsignedByte & 0xff;.
Now, unsignedValue can be checked in the range of 0 to 255. (signed would have values from -128 to 128)
OpenCV (CV_Type.*) Mat data type -----TO------ Java Array Types
CV_8U and CV_8S -> byte[] CV_16U and CV_16S -> short[] CV_32S -> int[] CV_32F -> float[] CV_64F-> double[]
Simple Example for thresholding on a grey image that converts from 8U which would be byte array to 16U1 so that we can avoid issues of signed byte arrays for Java
.....NOTE: input to this algorithm is expected to be gray of type CvType.CV_8U
//THRESHOLDING ALGORITHM int threshold = 100;
/*
SPECIAL NOTE
If CvType.CV_8U would use byte array BUT, warning java has signed not unsiged bytes
so pixel values would go from -128 to 128 and not the normal 0 to 255
byte[] imageB = new byte[(int) (gray.total() * gray.channels())];
gray.get(0, 0, imageB);
to avoid this first convert the Mat to 16UC1 and then create short[] array
gray.convertTo(gray, CvType.CV_16UC1);
short[] image = new short[(int) (gray.total() * gray.channels())];
gray.get(0,0, image);
If CvType.CV_32S use int array
If CvType.CV_32F use float array
If CvType.CV_64F use double array
IMPORTANT: it doesn't matter how many channels - gray = 1, color =3 the code below will appropriately
handle it. For example, we have an imageMat that is color and of type CvType.CV_8UC4
*/
//to avoid issues with Java's SIGNED byte array convert Mat from 8 to 16 and
//save in a short[] array
gray.convertTo(gray,CvType.CV_16UC1);
short[] image = new short[(int) (gray.total() * gray.channels())];
gray.get(0, 0, image);
//PROCESS the image --do a simple threshold
// if wanted to do it as row, col would write
int width = gray.width();
int height = gray.height();
int num_pixels = (int) gray.total();
int channels = gray.channels();
for (int r = 0; r < height; r++)
for (int c = 0; c < width; c++)
for (int i = 0; i < channels; i++) //in case color channels =3 otherwise for gray =1
{
//to get pixel corresponding to row r and column c
if (image[r * (width * channels) + c * channels + i] < threshold)
image[r * (width * channels) + c * channels + i] = 0;
else
image[r * (width * channels) + c * channels + i] = 255;
}
//put back the image
gray.put(0,0, image);
//seems we need to convert back to original gray for it to display correctly in JavaCameraView
gray.convertTo(gray,CvType.CV_8U);
return gray;
EXAMPLE 1 Slow -> faster comparison
It is okay to access pixels this way for small images or matrices with Mat.get/put.
JNI OVERHEAD CASE STUDY:
PLAY Youtube video to see difference in speed
Pixel manipulation the SLOW WAY
Pixel manipulation using Mat. get(row,col) and put(row, col, value) method. For example to threshold the imge
int threshold = 100;
//PROCESS the image --do a simple threshold
// if wanted to do it as row, col would write
int width = imageMat.width();
int height = imageMat.height();
int num_pixels = (int) imageMat.total();
double [] pixel = new double[3];
int channels = imageMat.channels();
int sum;
for (int r = 0; r < height; r++)
for (int c = 0; c < width; c++)
{ sum = 0;
pixel = imageMat.get(r,c);
for (int i = 0; i < 3; i++) //processing first 3 channels of rgb
{ //convert color to grey by averaging the 3 color values and dividing by 3
sum += pixel[i];
}
sum = sum /3;
for (int i = 0; i < 3; i++) //for color set only 3 color channels dont touch alpha channel
pixel[i] = sum;
imageMat.put(r,c,pixel);
}
Log.i("what", "wrong");
return imageMat;Pixel manipulation the FASTER WAY
If you want to manipulate the pixels on the Java side, perform the following steps:
- Allocate memory with the same size as the matrix in a byte array
- Put the image contents into that byte array using a single Mat.get call
- Manipulate the byte array contents.
- Make a single put call, copying the whole byte array to the matrix.
int threshold = 100;
/*
SPECIAL NOTE
If CvType.CV_8U would use byte array BUT, warning java has signed not unsiged bytes
so pixel values would go from -128 to 128 and not the normal 0 to 255
byte[] imageB = new byte[(int) (gray.total() * gray.channels())];
imageMat.get(0, 0, imageB);
to avoid this first convert the Mat to 16UC4 and then create short[] array
imageMat.convertTo(imageMat, CvType.CV_16UC4);
short[] image = new short[(int) (gray.total() * gray.channels())];
imageMat.get(0,0, image);
If CvType.CV_32S use int array
If CvType.CV_32F use float array
If CvType.CV_64F use double array
IMPORTANT: it doesn't matter how many channels - gray = 1, color =3 the code below will appropriately
handle it. For example, we have an imageMat that is color and of type CvType.CV_8UC4
*/
//to avoid issues with Java's SIGNED byte array convert Mat from 8 to 16 and
//save in a short[] array
imageMat.convertTo(imageMat, CvType.CV_16UC4);
short[] image = new short[(int) (imageMat.total() * imageMat.channels())];
imageMat.get(0,0, image);
//PROCESS the image --do a simple threshold
// if wanted to do it as row, col would write
int width = imageMat.width();
int height = imageMat.height();
int num_pixels = (int) imageMat.total();
int channels = imageMat.channels();
int sum;
for (int r = 0; r < height; r++)
for (int c = 0; c < width; c++)
{ sum = 0;
for (int i = 0; i < 3; i++) //processing first 3 channels of rgba - 4 channel color
{
//convert color to grey by averaging the 3 color values and dividing by 3
sum += image[r * (width * channels) + c * channels + i];
}
for (int i = 0; i < 3; i++) //for color set only 3 color channels dont touch alpha channel
image[r *(width*channels) + c*channels +i ] = (short) (sum / 3);
}
//put back the image calculated into the Mat gray object that is class variable
imageMat.put(0,0, image);
//seems we need to convert back to original gray for it to display correctly in JavaCameraView
imageMat.convertTo(imageMat,CvType.CV_8UC4);
Log.i("what", "wrong");
return imageMat;Example 2--- a fast way (here dropping blue value of rgb image)
A simple example that will iterate all image pixels and set the blue channel to zero,
which means that we will set to zero every element whose modulo is 3 equals zero, that is {0, 3, 6, 9, …}, as shown in the following piece of code:
public void filter(Mat image) { int totalBytes = (int)(image.total() * image.elemSize()); byte buffer[] = new byte[totalBytes]; image.get(0, 0,buffer); for(int i=0;i<totalBytes;i++){ if(i%3==0) buffer[i]=0; } image.put(0, 0, buffer); }First, we find out the number of bytes in the image by multiplying the total number of pixels (image.total) with the element size in bytes (image.elemenSize). Then, we build a byte array with that size. We use the get(row, col, byte[]) method to copy the matrix contents in our recently created byte array. Then, we iterate all bytes and check the condition that refers to the blue channel (i%3==0). Remember that OpenCV stores colors internally as {Blue, Green, Red}. We finally make another JNI call to image.put, which copies the whole byte array to OpenCV's native storage. An example of this filter can be seen in the following screenshot, which was uploaded by Mromanchenko, licensed under CC BY-SA 3.0:
