Problem with anti-aliasing with MSAA, drawing CSG with depth and FBO

Question

I updated OpenCSG for the modern version of OpenGL.

PixelFormatAttributes:

NSOpenGLPFAColorSize , 24 , NSOpenGLPFAAlphaSize , 8 , NSOpenGLPFADepthSize , 32 , NSOpenGLPFAStencilSize , 8 , NSOpenGLPFAAccelerated , NSOpenGLPFADoubleBuffer , NSOpenGLPFASupersample , NSOpenGLPFASampleBuffers, 1 , NSOpenGLPFASamples , 4 , 

FBO specifications: (tried to visualize FBO with multisample, but the lines became stronger and more visible, look at the screenshot below)
- created texture with power 2, GL_RGBA (checked GL_RGBA8 and GL_RGBA32F)
- GL_DEPTH24_STENCIL8 (checked GL_DEPTH32_STENCIL8, no result)

Just the Goldfeather algorithm:

 while (i < depth complexity) { take channel for render merge layers if no free channel render each layer with stencil func, mask and depth params to channel (FBO) } merge layers (taking texture from FBO and render objects again with applying shader below) 

In the shader, I have code for merging (the texture of the result from FBO overlaps over the rendering for testing, in OpenCSG it setupProjectiveTexture ):

  vec2 ndcPos = gl_FragCoord.xy / sizetexture.xy; vec4 maskColor = texture2D(maskTexture, ndcPos.xy); if (maskColor[channel] < 0.5) { discard; } 

It seems like after the FBO got an insufficiently clear texture or was not the right size.

EDIT:
These lines appear only in overlapping grid subtraction places.

EDIT 2:
Fixed with rendering for FBA without MSAA and applying FXAA by result.

Answer 1

Not sure if I understood your explanation correctly. What texture do you select in the shader?

I don’t know the specifics of the algorithm you are implementing, but if this texture contains the results of some previous calculations with a multisample already done, I’m really not sure if you can use this result for the whole fragment (and all its samples). The mask will fade around the edges, which seems like a problem here.

You may need to collect samples manually, for example. use sampler2DMS and read individual samples using texelFetch in the shader.

Another solution would be to set gl_SampleMask for all 1 in the first pass to prevent border fading.