Stencil using OpenGl ES 2.0

Question

I am trying to find a way to cut out a certain area of ​​the background texture so that against this background a certain screen does not appear on the screen. For instance:

This square can be any pattern. I use Frame Buffer Object and Stencil Buffer to achieve this effect. Here is the code:

fbo.begin(); //Disables ColorMask and DepthMask so that all the rendering is done on the Stencil Buffer Gdx.gl20.glColorMask(false, false, false, false); Gdx.gl20.glDepthMask(false); Gdx.gl20.glEnable(GL20.GL_STENCIL_TEST); Gdx.gl20.glStencilFunc(GL20.GL_ALWAYS, 1, 0xFFFFFFFF); Gdx.gl20.glStencilOp(GL20.GL_REPLACE, GL20.GL_REPLACE, GL20.GL_REPLACE); stage.getSpriteBatch().begin(); rHeart.draw(stage.getSpriteBatch(), 1); //Draws the required pattern on the stencil buffer //Enables the ColorMask and DepthMask to resume normal rendering Gdx.gl20.glColorMask(true, true, true, true); Gdx.gl20.glDepthMask(true); Gdx.gl20.glStencilFunc(GL20.GL_EQUAL, 1, 0xFFFFFFFF); Gdx.gl20.glStencilOp(GL20.GL_KEEP, GL20.GL_KEEP, GL20.GL_KEEP); background.draw(stage.getSpriteBatch(), 1); //Draws the background such that the background is not rendered on the required pattern, leaving that area black. stage.getSpriteBatch().end(); Gdx.gl20.glDisable(GL20.GL_STENCIL_TEST); fbo.end(); 

However, this does not work at all. How should I do this using Stencil Buffers? It is also difficult for me to understand glStencilFunc and glStencilOp. It would be very helpful if someone could shed light on these two.

UPDATE: I also tried to create something similar using glColorMask. Here is the code:

 Gdx.gl20.glClearColor(0, 0, 0, 0); stage.draw(); FrameBuffer.clearAllFrameBuffers(Gdx.app); fbo1.begin(); Gdx.gl20.glClearColor(0, 0, 0, 0); batch.begin(); rubber.draw(batch, 1); Gdx.gl20.glColorMask(false, false, false, true); coverHeart.draw(batch, 1); Gdx.gl20.glColorMask(true, true, true, false); batch.end(); fbo1.end(); toDrawHeart = new Image(new TextureRegion(fbo1.getColorBufferTexture())); batch.begin(); toDrawHeart.draw(batch, 1); batch.end(); 

This code produces the following: Instead of something like this: (Ignore window sizes and color tones)

Note. I am using the libgdx library.

Answer 1

When drawing before SpriteBatch state changes are ignored until end() called. If you want to use a stencil with SpriteBatch , you need to break the batch drawing. One thing, I quit FBOs, but that should not change.

 @Override public void create() { camera = new OrthographicCamera(1, 1); batch = new SpriteBatch(); texture = new Texture(Gdx.files.internal("data/badlogic.jpg")); texture.setFilter(TextureFilter.Linear, TextureFilter.Linear); TextureRegion region = new TextureRegion(texture, 0, 0, 256, 256); sprite = new Sprite(region); sprite.setSize(1f, 1f); sprite.setPosition(-0.5f, -0.5f); spriteUpsideDown = new Sprite(new TextureRegion(texture, 1f, 1f, 0f, 0f)); spriteUpsideDown.setSize(1f, 1f); spriteUpsideDown.setPosition(-0.5f, -0.5f); pattern = new Sprite(region); pattern.setSize(0.5f, 0.5f); pattern.setPosition(-0.25f, -0.25f); << Set Input Processor >> } 

The input processor allows you to set two boolean flags breakBatch1 and breakBatch2 using the keyboard (libgdx on the desktop), which are used to break the SpriteBatch drawing.

 @Override public void render() { Gdx.gl.glClearColor(1, 1, 1, 1); Gdx.gl.glClear(GL10.GL_COLOR_BUFFER_BIT | GL10.GL_STENCIL_BUFFER_BIT); batch.setProjectionMatrix(camera.combined); // setup drawing to stencil buffer Gdx.gl20.glEnable(GL20.GL_STENCIL_TEST); Gdx.gl20.glStencilFunc(GL20.GL_ALWAYS, 0x1, 0xffffffff); Gdx.gl20.glStencilOp(GL20.GL_REPLACE, GL20.GL_REPLACE, GL20.GL_REPLACE); Gdx.gl20.glColorMask(false, false, false, false); // draw base pattern batch.begin(); pattern.draw(batch); if(breakBatch1) { batch.end(); batch.begin(); } // fix stencil buffer, enable color buffer Gdx.gl20.glColorMask(true, true, true, true); Gdx.gl20.glStencilOp(GL20.GL_KEEP, GL20.GL_KEEP, GL20.GL_KEEP); // draw where pattern has NOT been drawn Gdx.gl20.glStencilFunc(GL20.GL_NOTEQUAL, 0x1, 0xff); sprite.draw(batch); if(breakBatch2) { batch.end(); batch.begin(); } // draw where pattern HAS been drawn. Gdx.gl20.glStencilFunc(GL20.GL_EQUAL, 0x1, 0xff); spriteUpsideDown.draw(batch); batch.end(); } 

Answer 2

Gdx.gl20.glStencilFunc(GL20.GL_REPLACE, GL20.GL_REPLACE, GL20.GL_REPLACE);

These are not valid glStencilFunc arguments. I think you mean glStencilOp here.

You need to use glGetError in your code, it will warn you about these errors.

Answer 3

I believe that your problem is that your initial stencil operation GL_REPLACE applies to all the drawn pixels in your rHeart.draw regardless of the shape of any texture used on the quadrant.

Thus, the stencil value is applied to each pixel of your ATVs, which gives you a problem.

If the texture used on your quadrant has an alpha channel, since GL_ALPHA_TEST not supported, you can set the shader to discard fully transparent pixels, preventing them from being drawn in the stencil buffer.