Using Pixel Buffer Objects (PBO's) on Android

Question

On Android, I try to do some OpenGL processing on camera frames, show these frames in the camera preview, and then encode the frames in the video file. I am trying to do this using OpenGL, using GLSurfaceView and GLSurfaceView.Renderer and with FFMPEG to encode the video.

I successfully processed the images using the shader. Now I need to encode the processed frames on the video. GLSurfaceView.Renderer provides the onDrawFrame method (GL10 ..). In this method, I try to read image frames using only glReadPixels (), and then put the frames in a queue for encoding on video. By itself, glReadPixels () is too slow - frame rate in one bit. I am trying to speed this up using Pixel Buffer Objects. This does not work. After pbo is connected, the frame rate remains unchanged. This is my first time using OpenGL, and I don't know where to start looking for a problem. Am I doing it right? Can someone give me some direction? Thanks in advance.

public class MainRenderer implements GLSurfaceView.Renderer, SurfaceTexture.OnFrameAvailableListener { . . public void onDrawFrame ( GL10 gl10 ) { //Create a buffer to hold the image frame ByteBuffer byte_buffer = ByteBuffer.allocateDirect(this.width * this.height * 4); byte_buffer.order(ByteOrder.nativeOrder()); //Generate a pointer to the frame buffers IntBuffer image_buffers = IntBuffer.allocate(1); GLES20.glGenBuffers(1, image_buffers); //Create the buffer GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, image_buffers.get(0)); GLES20.glBufferData(GLES20.GL_ARRAY_BUFFER, byte_buffer.limit(), byte_buffer, GLES20.GL_STATIC_DRAW); GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, image_buffers.get(0)); //Read the pixel data into the buffer gl10.glReadPixels(0, 0, this.width, this.height, GL10.GL_RGBA, GL10.GL_UNSIGNED_BYTE, byte_buffer); //encode the frame to video enQueueForEncoding(byte_buffer); //unbind the buffer GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, 0); } . . } 

Answer 1

As I recall, glBufferData() does not map your internal buffer to GPU memory, it just copies the data from your memory to the buffer (initializes).

To access the memory allocated by glBufferData() , you must use glMapBufferRange() . This function returns a Java Buffer object that you can read.

Answer 2

I have never tried something like this (opengl + video enconding), but I can say that reading from the memory of the SLOW device. Try double buffering, this might help, as the GPU can continue rendering in the second buffer while the DMA controller reads the data.

Download the profiler (check the GPU vendor for your device), this may give you some idea. Another thing that might help is to set the internal pbuffer format to something else, try decreasing the number and discarding the channel (alpha).

EDIT: if you feel like it, you can encode the video on a GPU that will increase, memory and processing wise, your application.