How to determine the direction of the robot from the image?

Question

I am developing a robot that can work in a corn factory and controlled by compass sensors, but I want to use the camera as the eye of a robot and use image processing to detect the angle of the motion error.

These are sample images.

processed image raw image segmented image

I use the next step

• Step 1: The current method I use will convert the color value to HSV, modified from this code

• Step 2: Thus, it will detect the selected color, which is brown or dirty, then I collect the left and right colors of brown or the selected color of each line of the image in two arrays (red dot).

• Step 3: I will build a 2 linear regression line as a blue dot and calculate the intersection point as a pink dot
• Step 4: Draw a green line to compare the pink dot with other images. I'm not sure what to do with this green line.
• The problem is that dirt or brown exists between the corn leaf, and then I force my code to skip the calculation

The question is how to filter the brown pixel between the corn leaf or another area that is not on the corn path? What algorithm or methodology should I study or apply to this problem?

EDIT1: using Spektre answer and looking better

Here is the result, after I applied it using JAVA + Boofcv

• Step 1: Threshold or Color Segmentation

• Step 2: Blur (use the Gauss and Med filter)

• Step 3: linear graph regression

Additional Information

Full code here

LinearRegression Class

10 sample images with the same process

Answer 1

For source image

I would:

• create a mask of brown material

  Just do H,S and ignore V You already have it. I use the integer 255 instead of the color ( Blue ) for the last calculation.

  

• blur mask

  This will remove small clusters of incorrectly selected parts. After that, you should mask the mask again, because the value of the mask will be slightly less than 255 if you do not get the fully selected areas. The larger the area, the greater the value (closer to 255 ). I support >=150

• scan mask with horizontal lines

• for each line, find the center of gravity of all selected pixels

  After blurring and reuse, the mask used is in Aqua . Therefore, calculate the midpoint x coordinate of all masked pixels in each row. This item is marked White .

• regression line through all centers of gravity

  I use my approximate search for this, but you can use whatever regression you want. The regressed line is marked with a red

  I used the linear equation x=x0+y*dx , where y=<0,pic1.ys> . And do a search at intervals:

   x0=<-pic1.xs,+2*pic1.xs> dx=<-10,+10> 

Where pic1.xs,pic1.ys is the resolution of the image. Thus, as you can see, I do not cover the full range of angles, but I think that this will not work on these extreme cases in any case (near horizontal directions). For such cases, you should do this instead of vertical lines and use x=y0+x*dy instead.

Here is C ++ . I did it with

  picture pic0,pic1; // pic0 - source img // pic1 - output img int x,y,h,s,v,px,pn,*p; color c; // copy source image to output pic1=pic0; pic1.save("cornbot0.png"); // create brown stuff mask for (y=0;y<pic1.ys;y++) // scan all H lines for (x=0;x<pic1.xs;x++) // scan actual H line { c=pic1.p[y][x]; // get pixel color rgb2hsv(c); // in HSV h=WORD(c.db[picture::_h]); s=WORD(c.db[picture::_s]); v=WORD(c.db[picture::_v]); // Treshold brownish stuff if ((abs(h- 20)<10)&&(abs(s-200)<50)) c.dd=255; else c.dd=0; pic1.p[y][x]=c; } pic1.save("cornbot1.png"); pic1.smooth(10); // blur a bit to remove small clusters as marked pic1.save("cornbot2.png"); // compute centers of gravity p=new int[pic1.ys]; // make space for points for (y=0;y<pic1.ys;y++) // scan all H lines { px=0; pn=0; // init center of gravity (avg point) variables for (x=0;x<pic1.xs;x++) // scan actual H line if (pic1.p[y][x].dd>=150) // use marked points only { px+=x; pn++; // add it to avg point pic1.p[y][x].dd=0x00004080; // mark used points (after smooth) with Aqua } if (pn) // finish avg point computation { px/=pn; pic1.p[y][px].dd=0x00FFFFFF;// mark it by White p[y]=px; // store result for line regression } else p[y]=-1; // uncomputed value } // regress line approx x0,dx; double ee; for (x0.init(-pic1.xs,pic1.xs<<1,100,3,&ee); !x0.done; x0.step()) // search x0 for (dx.init(-10.0 ,+10.0 ,1.0,3,&ee); !dx.done; dx.step()) // search dx for (ee=0.0,y=0;y<pic1.ys;y++) // compute actua solution distance to dataset if (p[y]!=-1) // ignore uncomputed values (no brown stuff) ee+=fabs(double(p[y])-x0.a-(double(y)*dx.a)); // render regressed line with Red for (y=0;y<pic1.ys;y++) { x=double(x0.aa+(double(y)*dx.aa)); if ((x>=0)&&(x<pic1.xs)) pic1.p[y][x].dd=0x00FF0000; } pic1.save("cornbot2.png"); delete[] p; 

I use my own class picture for images, so some members:

• xs,ys image size in pixels
• p[y][x].dd - pixel at position (x,y) as a 32-bit integer type
• p[y][x].dw[2] - pixel at position (x,y) as an integer type 2x16 bits for 2D fields
• p[y][x].db[4] - pixel at position (x,y) as an integer type 4x8 bits for easy access to the channel
• clear(color) - clears the entire image
• resize(xs,ys) - resize the image to a new resolution
• bmp - VCL Encapsulated GDI Bitmap with Canvas Access
• smooth(n) - fast blur n times

You can improve this with segmentation (removal of small clusters) depending on the area and position. You can also ignore too large peaks at avg points between neighbors. You can also detect the sky and ignore the entire area where the sky is present.

[edit1] smooth

It looks like this:

 void picture::smooth(int n) { color *q0,*q1; int x,y,i,c0[4],c1[4],c2[4]; bool _signed; if ((xs<2)||(ys<2)) return; for (;n>0;n--) { #define loop_beg for (y=0;y<ys-1;y++){ q0=p[y]; q1=p[y+1]; for (x=0;x<xs-1;x++) { dec_color(c0,q0[x],pf); dec_color(c1,q0[x+1],pf); dec_color(c2,q1[x],pf); #define loop_end enc_color(c0,q0[x ],pf); }} if (pf==_pf_rgba) loop_beg for (i=0;i<4;i++) { c0[i]=(c0[i]+c0[i]+c1[i]+c2[i])>>2; clamp_u8(c0[i]); } loop_end if (pf==_pf_s ) loop_beg { c0[0]=(c0[0]+c0[0]+c1[0]+c2[0])/ 4; clamp_s32(c0[0]); } loop_end if (pf==_pf_u ) loop_beg { c0[0]=(c0[0]+c0[0]+c1[0]+c2[0])>>2; clamp_u32(c0[0]); } loop_end if (pf==_pf_ss ) loop_beg for (i=0;i<2;i++) { c0[i]=(c0[i]+c0[i]+c1[i]+c2[i])/ 4; clamp_s16(c0[i]); } loop_end if (pf==_pf_uu ) loop_beg for (i=0;i<2;i++) { c0[i]=(c0[i]+c0[i]+c1[i]+c2[i])>>2; clamp_u16(c0[i]); } loop_end #undef loop_beg #define loop_beg for (y=ys-1;y>0;y--){ q0=p[y]; q1=p[y-1]; for (x=xs-1;x>0;x--) { dec_color(c0,q0[x],pf); dec_color(c1,q0[x-1],pf); dec_color(c2,q1[x],pf); if (pf==_pf_rgba) loop_beg for (i=0;i<4;i++) { c0[i]=(c0[i]+c0[i]+c1[i]+c2[i])>>2; clamp_u8(c0[i]); } loop_end if (pf==_pf_s ) loop_beg { c0[0]=(c0[0]+c0[0]+c1[0]+c2[0])/ 4; clamp_s32(c0[0]); } loop_end if (pf==_pf_u ) loop_beg { c0[0]=(c0[0]+c0[0]+c1[0]+c2[0])>>2; clamp_u32(c0[0]); } loop_end if (pf==_pf_ss ) loop_beg for (i=0;i<2;i++) { c0[i]=(c0[i]+c0[i]+c1[i]+c2[i])/ 4; clamp_s16(c0[i]); } loop_end if (pf==_pf_uu ) loop_beg for (i=0;i<2;i++) { c0[i]=(c0[i]+c0[i]+c1[i]+c2[i])>>2; clamp_u16(c0[i]); } loop_end #undef loop_beg #undef loop_end } } 

This is just an average weight of 3 pixels.

 (x,y)=(2*(x,y)+(x-1,y)+(x,y-1))/4 

and after that do the same with

 (x,y)=(2*(x,y)+(x+1,y)+(x,y+1))/4 

to avoid image bias. Then all this loops n times, and that’s all. You can ignore the clip and pixel parameters, in this case it is pf==_pf_rgba , but in any case it uses only one channel ... dec_color,enc_color just unpack, pack the color channels into / from the variable array to avoid problems with trimming and overflow on 8 bit channels, as well as to simplify / simplify the code (to support various pixels)

btw smooth base is the same as convolution with

 0.00 0.25 0.00 0.25 0.50 0.00 0.00 0.00 0.00 

and

 0.00 0.00 0.00 0.00 0.50 0.25 0.00 0.25 0.00 

Answer 2

If I am right, are you asking about the brown parts that are visible in the delusion or in other parts of the background?

How did you get the last image? I assume you multiplied the original image by the mask? Even if you haven’t done this, you can simply get the mask from the image by choosing the place where the image will be (any simple, very low threshold). (Apply an adaptive threshold value, an even more accurate version of the original, to get a better mask)

Take this mask and clean it with morphological operations, in your case closing will be enough. Morphology consists of many operations that can give you an extremely clean image mask. Read them.