SurfaceTexture.OnFrameAvailableListener ceases to be called

Question

I use the SurfaceTexture.OnFrameAvailableListener interface in my application, so I can use video frames as an OpenGL texture. Everything is set up as it should and works fine, but onFrameAvailable (SurfaceTexture surfaceTexture) stops calling after a few seconds, effectively and seemingly freezing the video in OpenGL, since no new texture is loaded via SurfaceTexture.updateTextImage.

I set a flag in onFrameAvailable to call updateTextImage from the GL stream and only if necessary. Currently, I set the flag to true with every callback, so the textured textures of the video are loaded every frame when the onFrameAvailable function is skipped. Like everything, everything works as it should, but it seems inefficient, since no new textured files need to be downloaded if they are still the same (movie frame).

AFAIK there are no memory leaks and logcat does not show any errors. In addition, the media player is configured to cycle, but the problem occurs before the completion of one run.

What can cause onFrameAvailable to no longer be called in a few seconds?

Answer 1

I saw a similar problem and debugged it. I, like you, had the Boolean flag, which indicated that one (or more!) Frames were ready for use.

The problem arose when I got two camera frames between two OpenGL frames (perhaps because Openraw processing in OpenGL was too slow). This meant that I set the Boolean flag twice. However, I then only read the data of this frame once, and it seems that updateTexImage has implemented some kind of queue function.

Replacing the boolean flag with an integer counter of pending camera frames solved the problem for me. Maybe this will work for you too?

(I suspect this is more efficient than just calling updateTexImage for each frame. At least in my code it was very rare (1-2%) for OpenGL frames long enough to span two camera frames.)

Answer 2

I had the same problem on some devices. Found a fix and thought that I would share. This is basically what @ user2254894 suggested, except that since the counter can be changed to two different threads, it’s nice to use 2 different wares. Here is a sample code:

private int _updateTexImageCounter = 0; private int _updateTexImageCompare = 0; 

and onFrameAvailable () is simple.

 @Override public void onFrameAvailable(SurfaceTexture surfaceTexture) { // increment every time a new frame is avail _updateTexImageCounter++; } 

Then in your GL update you will do something like the following ...

 public void update() { .... create texture... etc. .. // compare _updateTexImageCompare and _updateTexImageCounter if( _surfaceTexture!=null && _updateTexImageCompare != _updateTexImageCounter ) { // loop and call updateTexImage() for each time the onFrameAvailable() method was called below. while(_updateTexImageCompare != _updateTexImageCounter) { _surfaceTexture.updateTexImage(); _surfaceTexture.getTransformMatrix(x); _updateTexImageCompare++; // increment the compare value until it the same as _updateTexImageCounter } } } 

It worked for me. Let me know if there is a better way.