All in all, what I would do is first identify the grid, and then easily check if something is in that grid or not.
These are the following steps:
pseudo code:
xcount is array of int ycount is array of int for x=0 to width-1 do for y=0 to height-1 do foreach point do if point.x = x then xcount[x]++ if point.y = y then ycount[y]++
For your last image, the result would be something like this:
x-count:1,0,0,0,3,0,2,0,1,0,0,0,4,0,0,0,3,0,0,0,4 y-count:2,0,0,0,6,0,0,0,4,0,1,0,3,0,1,0,1
Now, to determine the size of the grid:
match = 0 for i = 1 - 10 do foreach xcount do if xcount mod i = 0 then matches [I] ++
Now we have an array containing an estimate (the number of matches) for 10 different mesh sizes. It might look something like this:
gridscores[] = 5,5,0,5,34,5,0,5 XgridSize = index of greatest gridSore
34 is undoubtedly the best match, and it is at index 5, so the grid size is 5.
Now that you know the size of the grid, you can easily find which points are not in this grid:
for each point wrongpoint = (point.x mod XgridSize! = 0) or (point.y mod YgridSize! = 0)
This works even if there are many wrong points. I did not describe in detail how to rotate and how to find the offset for the grid, but perhaps this will help you in the right direction.