Applying a Shader Filter to a BitmapData Object Ignores the Traversed Rectangle - How to Apply a Shader Filter Properly?

Question

I have the following code, made only for testing, as it was a kind of error that I wanted to nail:

_shader = new Shader(new TheShader() as ByteArray); _shader.data.width.value = [64.0]; _shader.data.height.value = [64.0]; _shaderFilter = new ShaderFilter(_shader); _sequence = new Vector.<BitmapData>(); var smallBD:BitmapData; var i:int; _delta = new Point(); var megabase:BitmapData = new TheBitmap().bitmapData; var _rect:Rectangle = new Rectangle(0, 0, 64, 64); for (i = 0; i < 64; i++) { smallBD = new BitmapData(64, 64, true, 0x00808080); //_rect.x = i; _rect.y = i; smallBD.applyFilter(megabase, _rect, _delta, _shaderFilter); _sequence.push(smallBD); } 

Then I loop through _sequence to see if changing the rectangle really does something. It does nothing if _shaderFilter is actually a shader filter. Testing with any of the built-in Flash filters works as intended, but with ShaderFilter it sometimes works as if the supplied rectangle was explicitly sourceBitmapData.rect, regardless of the source bitmap, and sometimes it behaves as if there was no data there wasn’t anymore, and the border is in a strange position - with a 512x384 bitmap, the edge of the area that is transferred to the shader is apparently in (256,192) or in the center of the bitmap. So far, I have managed to implement a workaround, that is, first copyPixels () the required area, and then applyFilter () in place. Can someone prove that this is a mistake, not me something is wrong?

PS: I use FlashDevelop for the purpose of the Flash Player 10.3 project, and I don't know if this fixes FP11.

Answer 1

Well, unfortunately, I can’t say how to fix it, but I can confirm that it’s not your fault!

The problem is that Flash ignores sourceRect completely when using custom shaders. At first I thought that this could pass values ​​to an undocumented parameter in the shader, but then I noticed that each pixel of the output bitmap changes even if sourceRect less or destPoint is non-zero. Also, there is no inCoord() function to match outCoord() , so it looks like this is not what the developers expected!

I can offer one suggestion; instead of copying the ROI to a new BitmapData object, add the float2 offset parameter to your shader and shift all pixel requests by this value. It will save some processing.

Here I used the test case to confirm the behavior:

ShaderTest.as:

 package { import flash.display.Sprite; import flash.display.Bitmap; import flash.display.BitmapData; import flash.display.Shader; import flash.geom.Point; import flash.geom.Rectangle; import flash.filters.ShaderFilter; final public class ShaderTest extends Sprite { [Embed(source="test.pbj",mimeType="application/octet-stream")] private static const sCopy : Class; final private function R( x, y, w, h ) : Rectangle { return new Rectangle( x, y, w, h ); } final public function ShaderTest( ) { super( ); var s : Shader = new Shader( new sCopy( ) ), f : ShaderFilter = new ShaderFilter( s ), d1 : BitmapData = new BitmapData( 256, 256, false, 0 ), d2 : BitmapData = new BitmapData( 128, 128, false ), b1 : Bitmap = new Bitmap( d1 ), b2 : Bitmap = new Bitmap( d2 ), w : Rectangle = R( 16, 16, 64, 64 ); b2.x = 274; addChild( b1 ); addChild( b2 ); for( var i : int = 0; i < 8; ++ i ) { for( var j : int = 0; j < 8; ++ j ) { d1.fillRect( R( i * 32 + 1, j * 32 + 1, 30, 30 ), (((i + j) & 1) * 0x00FF00) | (i << 21) | (j << 5) ); } } d2.applyFilter( d1, w, new Point( 10, 10 ), f ); d1.fillRect( R( wx, wy, 1, w.height ), 0xFF0000 ); d1.fillRect( R( wx, wy, w.width, 1 ), 0xFF0000 ); d1.fillRect( R( wx, wy + w.height - 1, w.width, 1 ), 0xFF0000 ); d1.fillRect( R( wx + w.width - 1, wy, 1, w.height ), 0xFF0000 ); } } } 

test.pbk:

 <languageVersion:1.0;> kernel bugtest <namespace:"Me";vendor:"Me";version:1;>{ input image4 src; output pixel4 dst; void evaluatePixel(){ dst = sampleNearest(src,outCoord()); } } 

Conclusion:

(the small square on the right copies from the large square using the shader. The red box shows sourceRect . destPoint - (10,10). Despite both of these settings, it actually displays the entire bitmap)