Assuming you have an alpha channel (like in a photo with a transparent background), it seems like a regular convolution blur matrix should suit you.
However, instead of going through the RGB channels, you only have to go through the ALPHA channel.
Check out the blur filter here: https://en.wikipedia.org/wiki/Kernel_%28image_processing%29
You are interested in Box Blur / Gaussian Blur. However, to make this effect smoother, you must use a larger matrix.
The reason that the algorithm will satisfy your needs is that if all the surrounding pixels have alpha-0, it will be 0. If 255, it will remain 255. Only the pixel in the border area between alpha 0/255 will be affected.
Edit:
Please check this script on chrome (in ff, which is very slow): http://jsfiddle.net/5L40ms65/
You can take a look at the algorithm at the end of the code. From the moment of implementation, I noted that: - there is no need to blur if all the neigbour pixels are 255 or 0 (alpha channel) - RGB blur is required in another case
Generally:
RADIUS = 2 (makes total width of matrix = 5) For x = 0..width for y = 0..width if all pixels in square of radius 2 are alpha = 0 do nothing elsif all pixels in square have alpha = 255 do nothing else pixel[x][y].RGB = average RGB of adjacent pixels where alpha != 0 pixel[x][y].ALPHA = average ALPHA in square
Example result with radius = 2
Of course, this is more of a conceptual program, there is a lot of space for memoization and customization of this script, however this should make the big picture transparent

source share