Processing and processing a zero copy of the camera on Android

Question

I need to make the reading process only on the processor side based on live camera data (only from the Y plane), and then visualize it on the GPU. Frames should not be displayed until processing is completed (therefore, I do not always want to display the last frame from the camera, only the last one that finished processing the processor). The rendering is separate from camera processing and has a target of 60 FPS, even if camera frames reach a lower speed than this.

There is a related, but higher question: The lowest overhead camera with a processor and graphical approach on android

Describe the current setting in more detail: we have a buffer pool of applications for camera data, where the buffers are “free”, “on display” or “pending display”. When a new frame comes from the camera, we get a free buffer, save the frame (or a link to it if the actual data is in some system pool), process and fix the results in the buffer, then set the buffer "waiting display". In the rendering stream, if there is any “pending display” buffer at the beginning of the rendering cycle, we click it instead of “display”, display the camera and render other content using the processed information calculated from the same camera frame.

Thanks to @fadden's answer on a related question, I now understand that the android camera2 API parallel output function uses shared buffers between different output queues, so it should not contain any copies of the data, at least on a modern android.

The comment suggested that I could simultaneously unload the outputs of SurfaceTexture and ImageReader and just “sit in the buffer” until the processing was complete. Unfortunately, I don’t think that this is applicable in my case because of the untied rendering that we still want to use at 60 FPS, and it will still need access to the previous frame while the new one is being processed to ensure that Something does not work out of sync.

One solution that came to mind was to have several SurfaceTextures - one in each of our buffers on the application side (we are currently using 3). With this scheme, when we get a new camera frame, we will get a free buffer from our application pool. Then we call acquireLatestImage() on the ImageReader to get the data to process and call updateTexImage() on the SurfaceTexture in a free buffer. During rendering, we just need to make sure that the SufaceTexture from the on-display buffer is GL-bound, and in most cases everything should be synchronized (since @fadden commented that there is a run between the call to updateTexImage() and acquireLatestImage() , but this the time window should be small enough to make it sparse, and perhaps it is possibly dectable and fixed using timestamps in buffers).

I note in the docs that updateTexImage() can only be called when the SurfaceTexture is attached to the GL context, which suggests that I need the GL context in the camera's processing thread so that the camera stream can do updateTexImage() on the SurfaceTexture in "free" the buffer, while the rendering stream can still display from the SurfaceTexture from the on-display buffer.

So, to the questions:

• Does this sound like a reasonable approach?
• Are SurfaceTextures primarily a light wrapper around a common buffer pool, or do they consume limited hardware and should be used sparingly?
• Is SurfaceTexture all cheap enough to cause that using multiple of them will still be a big win only when copying data?
• Is it planned to have two streams with different GL contexts with a different SurfaceTexture border, which can work in each of them, or am I asking about the world of pain and driver errors?

It sounds promising enough that I'm going to take it away; but thought it worth asking here if someone (mostly @fadden!) knows about any internal details that I missed that would make this a bad idea.

Answer 1

Interest Ask.

Background material

Having multiple threads with independent contexts is very common. Each application using hardware-accelerated rendering representation has a GLES context in the main thread, so any application using GLSurfaceView (or rolls its own EGL with SurfaceView or TextureView and an independent rendering stream) actively uses several contexts.

Each TextureView has a SurfaceTexture inside it, so any application that uses multiple TextureViews has multiple SurfaceTextures in a single thread. (In the structure in its implementation, which caused problems with several TextureViews, but it was a high-level problem, not a driver problem.)

SurfaceTexture, a / k / a GLConsumer, does not do much processing. When a frame comes from a source (in your case, a camera), it uses some EGL functions to "wrap" the buffer as an "external" texture. You cannot perform these EGL operations without an EGL context to work, so you need to bind SurfaceTexture to one, and why you cannot fit a new frame into a texture if the current context is current. You can see from the implementation of updateTexImage() that it does a lot of mysterious things with buffer queues and textures and fences, but none of them require copying pixel data. The only system resource that you really associate is the RAM, which is not insignificant if you shoot high-resolution images.

Connections

An EGL context can move between threads, but can only be "current" one thread at a time. Simultaneous access from multiple threads will require a lot of unwanted synchronization. This stream has only one "current" context. The OpenGL API evolved from single-threaded with a global state to multi-threaded, and instead of rewriting the APIs, they simply dragged the state to a local thread store ... hence the concept of "current".

You can create EGL contexts that share certain things between them, including textures, but if these contexts are in different streams, you have to be very careful when updating textures. Grafika is a good example of misuse .

SurfaceTextures are built on top of BufferQueues, which have a producer and consumer structure. The most interesting thing about SurfaceTextures is that they include both sides, so you can feed data in one direction and pull it out from the inside in one process (unlike, say, SurfaceView, where the consumer is far away). Like all Surface materials, they are built on top of IPC Binder, so you can feed the surface from one thread and safely updateTexImage() in another thread (or process). The API is designed so that you create a SurfaceTexture on the consumer side (your process), and then pass a link to the manufacturer (for example, a camera that mainly works in the mediaserver process).

Implementation

You will be causing a bunch of overhead if you constantly plug and unplug BufferQueues. Therefore, if you want to have three SurfaceTextures receiving buffers, you will need to connect all three to Camera2's output, and let them all receive a “buffer transfer”. Then you updateTexImage() around. Since SurfaceTexture BufferQueue is in “async” mode, you should always get the latest frame with every call, without having to “drop” the queue.

This layout was not really possible until the Lollipop era BufferQueue multi-mode output and Camera2 introduction changed, so I don't know if I tried this approach before.

All SurfaceTextures will be bound to the same EGL context, ideally in a thread other than the View UI thread, so you don't have to struggle with what is current. If you want to access the texture from the second context in another thread, you will need to use the SurfaceTexture attach / detash API , which explicitly supports this approach:

A new OpenGL ES texture object is created and populated with the SurfaceTexture image frame that was current the last time detachFromGLContext () was called.

Remember that switching contexts EGL is a user-side operation and does not affect the connection to the camera, which is a manufacturer-side operation. The overhead associated with moving a SurfaceTexture between contexts should be negligible - less than updateTexImage() , but you need to take the usual steps to ensure synchronization when communicating between threads.

Too bad ImageReader does not have a call to getTimestamp() , as this will greatly simplify the combination of buffers with the camera.

Conclusion

Using multiple SurfaceTextures to output a buffer is possible, but difficult. I see the potential advantage of the ping-pong buffer approach, where one ST is used to receive a frame in stream / context A, while another ST is used for rendering in stream / context B, but since you work in real I don’t think that this value is in extra buffering if you are not trying to cancel the time.

As always, it is recommended that you read the graphical architecture at the Android system level .